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PREFACE 


This  research  described  in  this  report  was  conducted  for  the  Air  Force  Research 
Laboratory,  Human  Effectiveness  Directorate,  Warfighter  Training  Research  Division 
(AFRL/HEA)  (formerly  known  as  Armstrong  Laboratory)  under  Work  Unit  2743-A3-04, 
Advanced  Instructional  Design  Advisor  (AIDA),  under  Contract  F41624-96-C-5006,  with  Mei 
Technology  Corporation.  The  Laboratory  Contract  Monitor  was  Dr  Robert  Yadrick, 
AFRL/HEAI,  at  Brooks  Air  Force  Base,  TX.  Dr  Yadrick  is  no  longer  with  AFRL/HEA  and  this 
report  was  submitted  for  publication  by  Dr  Donald  L.  Harville,  AFRL/HEAI. 

Documentation  of  this  research  was  initiated  at  Brooks  AFB  but  submission  for 
publication  was  delayed  due  to  personnel  reassignments  and  the  laboratory 
reorganization.  Therefore,  this  report  is  being  published  as  received  and  not  edited  by 
AFRL/HEA.  The  final  administrative  work  necessary  to  publish  this  report  was 
accomplished  by  the  Warfighter  Training  Research  Division  in  Mesa,  A Z. 
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TRAINING  INSTRUCTORS  TO  DEVELOP 
INTERACTIVE  MULTIMEDIA  COURSEWARE 


BACKGROUND 

This  report  describes  the  results  from  accomplishing  a  major  objective  of  the  Computer- 
based  Instruction  Component  of  the  State  Leadership  Consortium  for  Professional  Development. 
The  objective  called  for  instructors  from  community  colleges  across  the  state  to  be  trained  on  the 
use  of  the  Experimental  Advanced  Instructional  Design  Advisor  (XAIDA).  XAIDA  is  an 
instructional  development  system  that  generates  interactive  multimedia  courseware  (ICW). 

Results  from  training  instructors  to  develop  ICW  using  XAIDA  are  contained  in  this 
report.  XAIDA  is  a  system,  developed  under  the  sponsorship  of  the  Air  Force  Armstrong 
Laboratory  to  explore  techniques  for  automatically  generating  ICW.  XAIDA  consists  of  a 
program  called  Develop  for  entering  and  editing  lesson  topic  descriptions  and  a  program  called 
Deliver  for  presenting  instruction  on  the  topic.  Develop  provides  a  subject-matter  expert,  in  this 
case  a  community  college  instructor,  with  facilities  for  describing  the  structure  of  a  topic, 
describing  its  characteristics  and  behavior,  associating  multimedia  with  the  lesson  content,  and 
configuring  practice  exercises.  Deliver  provides  students  with  an  overview  of  the  subject  matter, 
a  detailed  presentation  of  lesson  material,  selective  review,  and  automatically  generated  practice 
exercises. 

METHODS 

XAIDA  training  was  conducted  in  two  workshops.  Both  training  workshops  were  three 
days  in  duration.  The  training  objectives  of  Workshop  1  were  to  be  able  to  use  XAIDA 
effectively  to  develop  ICW  that  describes  characteristics  of  a  topic  and  to  become  familiar  with 
the  XAIDA  interface  used  to  develop  ICW  that  teaches  theory  of  operation.  The  training 
objectives  for  Workshop  2  were  to  be  able  to  create  multimedia  (e.g.,  graphics,  audio,  and  video) 
for  XAIDA  courseware  and  be  able  to  use  XAIDA  to  develop  ICW  that  teaches  procedural  tasks. 
The  training  agendas  for  the  two  workshops  are  found  in  Appendix  A. 

Observations  were  made  throughout  training  on  changes  in  participants’  abilities  to 
develop  ICW  with  XAIDA,  self-reported  skills  level  using  XAIDA  and  computers  in  general, 
and  attitudes  toward  XAIDA.  Self-rated  skills  level  data  and  ICW  development  proficiency  data 
collected  from  the  three  veteran  participants  are  not  included  in  the  results,  due  to  obvious 
differences  in  their  level  of  experience  with  XAIDA.  The  veterans’  data  are  included  in  the 
results  that  address  attitudes  toward  XAIDA. 

PARTICIPANTS 

Thirteen  representatives  from  eleven  community  colleges  around  the  state  of  Texas 
participated  in  the  fist  workshop.  Eleven  representatives  attended  the  second  workshop.  Ten  of 
the  original  thirteen  representatives  were  new  to  XAIDA  and  three  were  returning  participants 
from  the  previous  year’s  project.  Two  of  the  ten  newcomers  attended  training  as  the  “colleague- 
to-be-trained.” 

Prior  to  training,  participants  were  surveyed  to  determine  how  often  they  use  computers 
and  their  level  of  familiarity  with  a  variety  of  software  applications.  Table  1  contains  percentages 
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of  participants  reporting  familiarity  with  the  software  applications.  Familiarity  with  a  Windows 
environment  and  PaintBrush  application  is  advantageous  to  learning  XAIDA,  but  not  necessary. 
Seventy-seven  percent  of  them  agreed  with  the  statement,  “I  regularly  use  a  computer  at  work.” 
Sixty-two  percent  of  participants  agreed  with  the  statement,  “I  regularly  use  a  computer  at 
home.” 


Table  1.  Percentage  of  Participants  Reporting  Familiarity  with  Software  Applications 


Software 

Percent  Familiar 

Microsoft  Word 

70 

Windows  Operating  System 

54 

Power  Point 

30 

Paint  Brush 

25 

Authorware® 

8 

RESULTS 

Abilities  Using  XAIDA 
Workshop  1 

The  most  direct  evidence  that  the  first  workshop’s  main  objective  was  achieved  is  the  12 
different  physical  characteristics  lessons  successfully  developed  during  training.  Lesson  topics 
are  listed  below.  The  *  indicates  veteran  developers.  Two  participants  from  different  colleges 
developed  lessons  on  the  Simpson  Meter. 


1. 

Respiratory  Sound  Sites 

7.  Netscape  Composer  Interface 

2. 

Software  Catalogues 

8.  Graph  Calculator 

3. 

Simpson  Meter  (2) 

9.  Road  Bicycle 

4. 

Computer  Hardware* 

10.  Nursing  Assessment  Process 

5. 

Linear  Equations* 

1 1 .  Self-contained  Breathing  Apparatus 

6. 

Infrared  Mouse* 

12.  Finance  Formula 

Workshop  2 

Following  instruction  on  developing  procedural  ICW,  participants  were  given  the  option 
to  work  on  any  of  the  three  lesson  types  (i.e.,  physical  characteristics,  theory  of  operation, 
procedural).  Three  of  the  1 1  opted  to  continue  work  on  their  physical  characteristics  lessons.  The 
most  direct  evidence  that  the  second  workshop’s  main  objective  was  achieved  in  part  is  the  eight 
different  procedural  lessons  developed  during  training.  Lesson  topics  are  listed  below.  The  * 
again  indicates  veteran  developers. 


1 .  Administering  a  Z-track  Intra-muscular  Injection 

5. 

Installing  RAM 

2.  Taking  Blood  Pressure* 

6. 

Administering  Oral  Medication 

3.  Creating  A  Webpage 

7. 

Donning  Self-contained  Breathing  Apparatus 

4.  Chain  Rule 

8. 

Balancing  a  Bank  Book 
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Self-perceived  Computer  Skills  Levels 

P aper-and-pencil  measures  of  self-perceived  computer  skills,  including  newly  acquired 
skills  using  XAIDA,  were  taken  before  and  after  the  workshops.  The  results  of  these  self-report 
data  are  presented  in  Figure  1.  Participants  rated  their  skills  levels  on  a  four-point  scale  where 
“1”  represented  none,  “2”  represented^,  “3”  represented  good,  and  “4  represented  expert.  The 
computer  skill  of  using  a  database  was  only  measured  at  the  first  workshop,  indicated  as  (SI)  in 
the  figure  below.  Also  the  computer  skill  of  creating  audio  was  only  measured  at  the  second 
workshop,  indicated  as  (S2)  in  the  figure  below.  Significant  increases  in  self-perceived  skills 
levels  were  found  with  the  following  computer  skills:  using  XAIDA  Develop  (program  for 
creating  ICW),  using  XAIDA  Deliver  (program  for  presenting  ICW  to  students),  managing  files, 
managing  directories,  creating  graphics,  using  the  scanner,  and  using  DOS.  Computer  skills 
unrelated  to  the  training  experience,  excluding  evaluation  tasks,  showed  no  change. 


using  a 
database  (SI) 

creating  audio 
(S2) 

using  DOS* 


scanning* 


creating  video 


programming 

word 

processing 

typing 

managing 

directories* 

managing  files* 

creating 

graphics* 

using  XAIDA 
Deliver* 

using  XAIDA 
Develop* 


1  1.5  2  2.5  3  3.5  4 


■  Before  Workshop  1 
B  After  Workshop  2 


Figure  1.  Mean  Ratings  of  Computer  Skills  of  New  Participants  Before  and  After  Workshops  (N=8) 

[*  Indicates  a  significant  (g  <  .05)  gain  in  skill  level  after  training.] 
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Participants’  rated  their  levels  of  proficiency  developing  ICW  with  XAIDA.  The  results 
from  both  3 -day  workshops  are  represented  in  the  next  figure.  Participants  rated  their  proficiency 
on  a  10-point  scale  where  “1”  represented  a  novice  user  and  “10”  represented  an  expert  user  of 
XAIDA  Develop.  Significant  increases  in  perceived  levels  of  proficiency  were  found  at  the  end 
of  the  first  two  days  of  training,  regardless  of  the  training  workshop.  The  first  two  days  were 
spent  learning  new  skills.  At  the  end  of  the  final  days  of  training,  no  significant  increases  in 
proficiency  were  reported.  The  final  day  of  the  first  workshop  was  spent  introducing  participants 
to  the  theory  of  operation  interface.  They  did  not  have  an  opportunity  to  use  the  interface  to 
develop  their  own  lessons.  The  final  day  of  the  second  workshop  was  spent  developing 
procedural  lessons  and  completing  physical  characteristics  lessons. 

expert  o 

9 
8 
7 
6 
5 
4 
3 
2 

NOVICE  1  - , - , - - 

Before  training  End  of  Day  1  End  of  Day  2  End  of  training 


Figure  2.  Mean  Ratings  of  Proficiency  with  XAIDA  Develop  for  New  Participants  Across  Workshops  (N  =  9) 


On  average  participants  rated  themselves  3.3  points  higher  on  the  10-point  scale  after  the 
first  workshop  compared  to  before  the  first  workshop.  This  increase  was  statistically  significant 
(t  (7)  =  5.3,  p  <  .001).  On  average  participants  rated  themselves  1.8  points  higher  on  the  10-point 
scale  after  the  second  workshop  compared  to  before  the  second  workshop.  This  increase  was  also 
statistically  significant  (t  (7)  =  5.0,  p  <  .002).  The  second  workshop  was  found  to  have  an  overall 
positive  effect  on  participants’  proficiency  levels  (M  End  of  Workshop  i  =  4.3,  M  End  of  workshop  2  =  5.4, 

E  difference  ^  .05). 

Participant  Attitudes  toward  XAIDA 

Acceptance  and  willingness  of  personnel  to  adopt  new  technology  is  imperative  to 
organizations  planning  to  successfully  integrate  new  technology.  Therefore,  an  exploration  was 
made  of  the  participants’  attitudes  toward  XAIDA.  Across  the  two  workshops,  participants 
responded  to  open-ended  items  asking  for  their  impressions  of  XAIDA.  Participants’  comments 
to  the  items  were  collected  before  training  and  at  the  end  of  the  first,  second,  and  final  day  of 
training,  during  both  workshops.  Participants’  comments  were  coded  into  seven  categories  that 
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have  been  found  to  reflect  users’  concerns  when  adopting  educational  innovations  (Hall  1979- 
Hall,  George,  &  Rutherford,  1986;  Bailey  &  Palsha,  1992).  Comments  were  further  coded  as  ’ 
positive,  negative,  and  neutral  statements.  Results  are  presented  in  Figure  3  and  Figure  4.  Users 
who  become  accepting  of  an  innovation  tend  to  express  positive  concerns  that  fall  into  the  higher 
categories  -consequences,  collaboration,  and  refocusing.  "" 

The  seven  categories  and  examples  of  coding  criteria  follow: 

1 .  Awareness-  have  or  have  not  heard  of  it,  interested  in  the  area; 

2.  Information-  limited  knowledge  about  it,  what  are  the  possible  applications; 

3.  Personal—  effects  on  professional  status,  how  my  tasks  will  change; 

4.  Management-  coordination  of  tasks  and  people,  conflict  between  interests  and 

responsibilities; 

5.  Consequences  how  it  affects  classroom  training,  its  application  to  training; 

6.  Collaboration  --coordinating  efforts  with  others,  help  others  with  the  innovation;  and 

7.  Refocusing- revise  the  approach,  modify  based  on  use. 

Two  different  patterns  of  concerns  emerged  across  the  workshops.  The  pattern  found  in 
the  first  workshop  revealed  that  participants  came  to  training  looking  for  more  information  about 
XAIDA.  During  the  course  of  the  first  workshop,  comments  made  were  of  the  personal  nature. 
Concerns  shifted  to  the  higher  level  categories  as  the  first  workshop  ended.  Participants  then 
expressed  as  many  personal  as  consequence  concerns. 

The  pattern  of  concerns  found  at  the  beginning  of  the  second  workshop  suggests  that 
participants  began  considering  what  ICW  had  to  offer  them  and  their  colleagues.  Day  1  of  the 
second  workshop  involved  learning  how  to  create  multimedia,  which  raised  management 
concerns — equipment  needs  and  technical  support.  Day  2  involved  developing  procedural  ICW. 
This  training  activity  stimulated  personal  concerns.  At  the  end  of  both  workshops,  over  half  of 
participants’  concerns  addressed  consequences,  collaboration,  and  refocusing  issues.  This  pattern 
indicates  that  participants  are  likely  to  be  early  adopters  of  new  technology. 


■  Refocusing 

■  Collaboration 

■  Consequences 
13  Management 

□  Personal 

■  Information 
■Awareness 


Figure  3.  Proportion  of  Comments  in  Each  Concern  Category  across  Workshops 
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Figure  4.  Proportion  of  Positive,  Negative,  and  Neutral  Comments  about  XAIDA  across  Workshops 

Overall  the  majority  of  participants’  comments,  as  shown  in  Figure  4,  were  positive.  The 
increases  in  negative  comments  are  largely  due  to  participants  developing  ICW  on  their  own. 
Developing  ICW  with  XAIDA  required  participants  to  think  differently  about  subject  matter  for 
which  they  possessed  expertise.  XAIDA  requires  that  you  think  about  how  to  describe  a 
characteristic,  rule  of  behavior,  or  procedural  step,  not  how  to  teach  it.  The  fact  that  the  majority 
of  participants’  comments  were  positive  further  indicates  that  participants  are  behaving  as  early 
adopters  of  new  technology. 

Appendix  B  contains  randomly  selected  comments  made  by  participants  during  the  first 
and  second  workshops,  respectively. 

DISCUSSION 

Each  instructor  who  completes  the  workshops  possesses  the  skills  necessary  to  develop 
ICW.  Each  is  capable  of  developing  ICW  that  teaches  the  structure  of  a  topic  and  its 
characteristics,  how  to  reason  about  a  topic,  procedures  associated  with  a  topic,  and  multimedia 
to  communicate  information  about  the  topic. 

Ten  of  the  instructors  completed  their  ICW  modules  in  preparation  for  classroom- 
evaluations.  The  next  step  was  preparation  for  the  classroom  evaluations.  Eight  of  the  ten 
instructors  incorporated  their  courseware  modules  into  their  curriculum.  Two  instructors  did  have 
access  to  students  during  the  course  of  the  project. 
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APPENDIX  A:  WORKSHOP  1  AND  WORKSHOP  2  TRAINING  AGENDAS 
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Workshop  1  Training  Agenda 


Introduction  by  Linda  Rivas 

Welcome  to  Training 

A.  Discuss  of  the 
research  and 
development  project 

B.  Brief  introduction  on 
what  XAIDA  is  and 
what  XAIDA  does 

Pre-training  assessment 

View  physical  characteristics 
and  theory  of  operation  lessons 
in  XAIDA  Deliver 


Review  Lesson  Introduction  and 
Lesson  Body  interfaces/editors 


Begin  developing  own  physical 
characteristics  lesson 


Presentation  of  theory  of 
operation  lesson 

View  theory  of  operation 
lesson  in  XAIDA  Deliver 

Present  theory  of  operation 
interfaces/editors  in  XAIDA 
Develop 

Pair-up  to  develop  theory  of 
operation  practice  lesson  from 
tutorial 


Presentation  of  Lesson 
Introduction  interfaces/editors 
in  XAIDA  Develop  for  physical 
characteristics 


Begin  developing  a  physical 
characteristics  practice  lesson 

Presentation  of  Lesson  Body 
interfaces/editors  in  XAIDA 
Develop 

Continue  developing  the 
physical  characteristics  practice 
lesson 


Continue  lesson  development 

Present  Practice 
interfaces/editors  in  XAIDA 
Develop 

Show-and-tell 


Documentation  for 
Reimbursement  session  by 
Linda  Rivas 

Review  theory  of  operation 
interfaces/editors 

Continue  developing  theory  of 
operation  practice  lesson 


Discuss  lesson  topics  and 
materials  brought  for  lesson 
development 

Distribute  XAIDA  user  manuals 


End  of  day  assessment 


End  of  day  assessment 


Post-training  assessment 
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Workshop  2  Training  Agenda 


Day  1  > 

;•  « wBmmmmm  m 

Pre-training  assessment 

Review  XAIDA  editors 

Interested  in  using  Theory  of 
Operation? 

Developing  procedural  lessons 

Work  on  lesson  type  of  choice 

Learn  how  to  create  multimedia 

Introduction  to  procedural  lesson 
View  example  XAIDA 
procedural  lesson 

Pre-plan  procedural  lesson 

Gather  and  create  lesson  media 

Continue  developing  lessons 

A  moment  with  Linda! 

Continue  developing  lessons 

Setup  dates  for  classroom 
evaluation 

End  of  day  assessment 

End  of  day  assessment 

Post-training  assessment 
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Workshop  1:  Comments  about  XAIDA 


Before  Training 

End  of  Day  1 

End  of  Day  2 

After  Training 

Good  development  tool  for 
tutorials.  Easy  to  design  lessons. 
Easy  to  add  resources.  Easy  to 
navigate. 

I'm  impressed  but 
implementation  will  be  a  big  step. 
We  currently  have  29  courses. 
Each  course  is  96  hours  in  length 
and  we  have  3  instructors  who 
currently  work  10-12  hours  per 
day  at  work  and  then  do  the  UP 
at  home.  1  will  do  everything  1  can 
to  bring  into  classroom  however 
time  is  critical. 

Impressive.  1  hope  it  will  work  for 
our  program. 

1  think  it  is  a  good  tool  we  can 
use  on  a  permanent  basis.  1  hope 
that  we  could  develop  a  series  of 
lessons. 

Sounds  good-l’m  ready  to  see  it! 

This  version  has  more  features 
than  the  previous  version  1  used. 

It  is  a  very  interesting  software 
package.  1  am  anxious  to  learn 
how  flexible  it  will  be  to  adapt  to  a 
broad  variety  of  tutorials. 

More  intuitive  than  the  older 
versions.  Still  restrictive  for 
possible  lesson  topics.  1  think  it 
will  take  much  less  time  to 
develop  a  lesson  with  this 
version. 

XAIDA  seems  powerful  in  that  it 
does  a  lot  of  the  development 
work  for  you.  The  interface  is  not 
as  intuitive  as  1  would  like  to  see. 
Some  points  require  buttons  that 
are  not  exactly  prominent. 

Still  up  for  grabs.  1  feel  it  is  going 
to  take  time  and  practice  to  get 
comfortable  but  it  will  happen  if 
you  all  stay  with  us. 

1  am  a  little  more  comfortable 
than  yesterday.  But  have  a  long 
way  to  go.  1  am  not  going  to  say 
that  this  is  user  friendly  since  I’m 
not  familiar  with  like  programs. 

It  should  be  a  useful  resource  to 
me.  It  imposes  some  restrictions 
on  presentation;  however,  none 
that  1  can  see  that  can’t  be 
overcome. 

I'm  still  very  high  on  it!  I’ve 
learned  a  lot  today,  and  hope  to 
learn  more  tomorrow.  I've  come 
a  long  way  but  have  miles  to  go 
before  1  feel  comfortable  with  it. 

Pretty  big!!  1  think  1  can  get  it  in 
time 

Good  tool  to  use,  very  easy  to 
put  together! 

A  very  useful  program  for 
teaching  a  physical  description. 

Very  useful  for  practice  with 
questions. 

Busy  interface,  not  very  intuitive. 

1  feel  I’ve  got  a  fairly  good  grasp 
on  this  because  it’s  what  we 
worked  with  in  most  of  the 
workshop. 

-Very  good — straightforward  if 
the  topic  can  be  placed  in  a 
physical  paradigm. 

Complex,  yet  simple— still  need 
to  practice. 

It  can  be  useful  for  several 
different  fields. 

Workshop  2:  Comments  about  XAIDA 


Begin  Training 

End  of  Day  1 

End  of  Day  2 

End  of  Training 

Very  good  for  tutorials. 

Parts  identification  has  limited 
use  in  mathematics;  procedural 
lessons  should  be  much  more 
useful. 

1  think  it  will  greatly  enhance 
learning  inside  or  outside  the 
classroom.  1  went  through  the 
student  program  that  1  developed 
and  felt  that  it  was  user  friendly 
and  interesting. 

My  problems  are  creating  the 
questions  and  getting  resources 
into  the  lesson.  I’ve  spent  most  of 
my  time  doing  (attempting)  these 
tasks. 

Still  very  impressive,  however, 
not  too  sure  how  or  if  it  will  be  so 
user-friendly  to  my  colleagues — 
they  question  how  to  use  it  in  the 
academic  setting,  i.e.,  English, 
history,  government. 

1  think  this  is  an  excellent  teaching 
tool  for  the  classroom.  It  is  easy  to 
operate  (as  a  student)  and  very 
interesting  using  the  multimedia.  1 
need  lots  of  technical  support  but 
am  very  impressed  with  the 
program. 

Procedural  lessons  will  be  more 
useful  for  multimedia  and 
multimedia  will  make  the  program 
more  interesting. 

Appears  to  have  far  more  utility 
than  what  we  learned  about  it  in 
December. 

After  seeing  a  procedural  lesson,  it 
seems  to  possess  some  utility  in 
adding  a  visual  element  to  the 
learning  process.  Creating  the 
multimedia  elements  is  fun. 

Multimedia  really  enhances  the 
XAIDA  lesson. 

Equipment  will  be  needed  at  our 
level. 

It’s  going  to  be  good,  but  now  1 
need  to  change  some  of  the 
“canned’’  statements  as  that 
they  will  apply  to  procedures 
rather  than  parts. 

Good  program. 

1  haven’t  yet,  but  1  know  this  will 
be  areat! 

XAIDA  can  probably  be  used  for 
math  procedures. 

Cool  stuff! 

1  feel  1  understand  it  better, 
however,  implementing  still 
seems  to  be  a  problem  for  me. 

1  spent  a  lot  of  time  setting  up 
bitmaps  for  the  presentation.  1 
really  did  not  progress  far. 

1  like  it  more. 

It  really  was  great.  Now  1  feel  so 
much  better.  This  program  can 
be  used  if  given  a  chance,  even 
by  English/History. 

Still  convoluted,  counter  intuitive, 
hard  to  use.  More  trouble  than  it's 
worth,  for  my  subject. 

Powerful,  flexible,  still  complex, 
but  extremely  usable. 

A  good  program  for  my  field.  The 
trend  in  the  safety  field  is  CBT.  1 
plan  to  go  back  and  put  several 
training  programs  on  XAIDA. 

Well,  1  think  this  will  have  a  lot  of 
uses.  I’m  looking  forward  to 
going  back  to  my  campus  and 
introducing  the  program  to  other 
faculty  and  students. 
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